32 research outputs found
Early Turn-taking Prediction with Spiking Neural Networks for Human Robot Collaboration
Turn-taking is essential to the structure of human teamwork. Humans are
typically aware of team members' intention to keep or relinquish their turn
before a turn switch, where the responsibility of working on a shared task is
shifted. Future co-robots are also expected to provide such competence. To that
end, this paper proposes the Cognitive Turn-taking Model (CTTM), which
leverages cognitive models (i.e., Spiking Neural Network) to achieve early
turn-taking prediction. The CTTM framework can process multimodal human
communication cues (both implicit and explicit) and predict human turn-taking
intentions in an early stage. The proposed framework is tested on a simulated
surgical procedure, where a robotic scrub nurse predicts the surgeon's
turn-taking intention. It was found that the proposed CTTM framework
outperforms the state-of-the-art turn-taking prediction algorithms by a large
margin. It also outperforms humans when presented with partial observations of
communication cues (i.e., less than 40% of full actions). This early prediction
capability enables robots to initiate turn-taking actions at an early stage,
which facilitates collaboration and increases overall efficiency.Comment: Submitted to IEEE International Conference on Robotics and Automation
(ICRA) 201
Gaze, Posture and Gesture Recognition to Minimize Focus Shifts for Intelligent Operating Rooms in a Collaborative Support System
This paper describes the design of intelligent, collaborative operating rooms based on highly intuitive, natural and multimodal interaction. Intelligent operating rooms minimize surgeon’s focus shifts by minimizing both the focus spatial offset (distance moved by surgeon’s head or gaze to the new target) and the movement spatial offset (distance surgeon covers physically). These spatio-temporal measures have an impact on the surgeon’s performance in the operating room. I describe how machine vision techniques are used to extract spatio-temporal measures and to interact with the system, and how computer graphics techniques can be used to display visual medical information effectively and rapidly. Design considerations are discussed and examples showing the feasibility of the different approaches are presented
Seeing Beyond: Real-time Ultrasound Image Integration in Augmented Reality Based Telementoring
Ultrasound imaging, when aptly integrated with augmented reality based medical telementoring, may be beneficial as an assistive tool in a range of trauma procedures including removal of foreign objects from blast injuries and central or peripheral venous access. Expected benefits include reduced procedure completion time, higher efficiency, and higher incision accuracy. This paper describes the implementation strategy selected for the integration of real time ultrasound imaging in the trainee view of a telementoring system. The proposed strategy augments the view of the trainee surgeon by displaying the ultrasound image directly below and parallel to the ultrasound transducer. The developed system features a fiducial marker based tracking approach employing a triplanar geometric fixture. An experiment was designed to demonstrate the system function and validate its performance
Eye-Tracking Metrics Predict Perceived Workload in Robotic Surgical Skills Training
Objective:
The aim of this study is to assess the relationship between eye-tracking measures and perceived workload in robotic surgical tasks.
Background:
Robotic techniques provide improved dexterity, stereoscopic vision, and ergonomic control system over laparoscopic surgery, but the complexity of the interfaces and operations may pose new challenges to surgeons and compromise patient safety. Limited studies have objectively quantified workload and its impact on performance in robotic surgery. Although not yet implemented in robotic surgery, minimally intrusive and continuous eye-tracking metrics have been shown to be sensitive to changes in workload in other domains.
Methods:
Eight surgical trainees participated in 15 robotic skills simulation sessions. In each session, participants performed up to 12 simulated exercises. Correlation and mixed-effects analyses were conducted to explore the relationships between eye-tracking metrics and perceived workload. Machine learning classifiers were used to determine the sensitivity of differentiating between low and high workload with eye-tracking features.
Results:
Gaze entropy increased as perceived workload increased, with a correlation of .51. Pupil diameter and gaze entropy distinguished differences in workload between task difficulty levels, and both metrics increased as task level difficulty increased. The classification model using eye-tracking features achieved an accuracy of 84.7% in predicting workload levels.
Conclusion:
Eye-tracking measures can detect perceived workload during robotic tasks. They can potentially be used to identify task contributors to high workload and provide measures for robotic surgery training.
Application:
Workload assessment can be used for real-time monitoring of workload in robotic surgical training and provide assessments for performance and learning
Augmented Reality Future Step Visualization for Robust Surgical Telementoring
Introduction
Surgical telementoring connects expert mentors with trainees performing urgent care in austere environments. However, such environments impose unreliable network quality, with significant latency and low bandwidth. We have developed an augmented reality telementoring system that includes future step visualization of the medical procedure. Pregenerated video instructions of the procedure are dynamically overlaid onto the trainee's view of the operating field when the network connection with a mentor is unreliable.
Methods
Our future step visualization uses a tablet suspended above the patient's body, through which the trainee views the operating field. Before trainee use, an expert records a “future library” of step-by-step video footage of the operation. Videos are displayed to the trainee as semitransparent graphical overlays. We conducted a study where participants completed a cricothyroidotomy under telementored guidance. Participants used one of two telementoring conditions: conventional telestrator or our system with future step visualization. During the operation, the connection between trainee and mentor was bandwidth throttled. Recorded metrics were idle time ratio, recall error, and task performance.
Results
Participants in the future step visualization condition had 48% smaller idle time ratio (14.5% vs. 27.9%, P < 0.001), 26% less recall error (119 vs. 161, P = 0.042), and 10% higher task performance scores (rater 1 = 90.83 vs. 81.88, P = 0.008; rater 2 = 88.54 vs. 79.17, P = 0.042) than participants in the telestrator condition.
Conclusions
Future step visualization in surgical telementoring is an important fallback mechanism when trainee/mentor network connection is poor, and it is a key step towards semiautonomous and then completely mentor-free medical assistance systems
Human posture recognition for intelligent vehicles
The article of record as published may be found at http://dx.doi.org/10.1007/s11554-010-0150-0Pedestrian detection systems are finding their
way into many modern ‘‘intelligent’’ vehicles. The body
posture could reveal further insight about the pedestrian’s
intent and her awareness of the oncoming car. This article
details the algorithms and implementation of a library for
real-time body posture recognition. It requires prior person
detection and then calculates overall stance, torso orientation in four increments, and head location and orientation,
all based on individual frames. A syntactic post-processing
module takes temporal information into account and
smoothes the results over time while correcting improbable
configurations. We show accuracy and timing measurements for the library and its utilization in a training
application.Office of Naval Researc
Wisdom in Our Fingers—Or How Embodied Interaction Can Shape Future Work
Applications of body- and gesture-based interfaces, and recognition in human-machine systems, are increasingly being used in healthcare, especially the use of robots in operating rooms